Method and Apparatus for Generating Multiple Image Views for a Multiview Autosteroscopic Display Device

ABSTRACT

A method for generating multiple synthesized images for a multiview autostereoscopic display device is disclosed. The method comprises the steps of:
         a) providing a stereoscopic input image composed of a left input image and a right input image,   b) determining respective pixel pairs of the left and right input images,   c) generating disparity information from the respective pixel pairs of the left and right input images in the form of a first disparity map,   d) scaling each disparity value of said first disparity map so that the range of the scaled disparity values reaches at least one of the boundaries of a preset maximum disparity range associated with said display device, thereby generating a second disparity map, and   e) based on said second disparity map, generating a preset number of synthesized images for simultaneously displaying on said multiview autostereoscopic display device, wherein each pair of adjacent synthesized images presents a synthesized stereoscopic image.

TECHNICAL FIELD

The present invention relates generally to multiview autostereoscopicdisplay devices. More specifically, the present invention relates to amethod and an apparatus for generating multiple image views for amultiview autostereoscopic display device.

BACKGROUND ART

Autostereoscopy is a well-known method of displaying three-dimensionalimages that can be viewed without the use of special headgear orglasses. Autostereoscopic displays have been developed to produce 3-Dstill or video images visible to the unaided eye. Currently, severaltechnologies exist for autostereoscopic 3D displays, including theflat-panel solutions which are regarded as the most promising technologyin this field. Such flat-panel 3D displays employ lenticular lenses orparallax barriers that redirect incoming imagery to several viewingregions at a lower resolution. In such displays, to achieve thedifferent 3D images for the different viewing zones, the incoming imageis split into a multitude of views corresponding to different viewingangles. These views are spliced into a 3D image and an array ofcylindrical lens focuses each view into different directions. Theangular separation between adjacent views is designed such that within aspecified viewing distance from the display, the viewer will perceive adifferent image with each eye, giving a stereo image. Such displays canhave multiple viewing zones allowing multiple users to view the image atthe same time. In general, the number of viewing zones of such displaysranges from five to nine, but multiview autostereoscopic displays withmore than ten views have also been investigated. Various literaturedescribes the principles and technology of multiview autostereoscopicimaging, for example U.S. Pat. No. 6,064,424.

Although the most realistic multiview imaging may obtained by usingmultiple cameras capturing images for the respective views to bedisplayed, such camera configurations are expensive and also conflictingwith space limitations in most case. A common solution to eliminate theneed of the high number of cameras is the use of a single pair of camerafor capturing a stereo image and subsequent computer image processing tosynthesize the required number of additional (artificial) views for themultiview display. Such a technology is disclosed, for example, in U.S.Pat. No. 6,366,281, which relates to a panoramagram and a method formaking the same from a pair of stereoscopic source images. The pair ofsource images is, for example, a left image and a right image havingdifferent perspective views which are spaced apart in a horizontalplane. The left image may be considered the starting image and the rightimage the ending image, or vice versa. Control points are defined atcorresponding locations in the source images, and each control pointincludes position and colour information. A plurality of intermediateimages are created by the so called “morphing” process from the startingimage to the ending image using position and colour information fromeach of the corresponding locations. Preferably, the morphing processinvolves creating transformations of the source images based on thephysical proximity from each control point; in other words, thetransformation is affected strongly by control points which are nearbyand is less affected by control points which are far away. Thus acontinuous panoramagram is produced synthetically from the two originalviews, namely the left camera view and the right camera view. Theintermediate images and the source images are then interdigitated tocreate a single output image with a continuum of views ranging from theoriginal left image to the original right image. This solution has thedrawback that the depth effect of the original stereo image is reducedin the adjacent synthesized views by the morphing between the originallycaptured left and right images, and therefore the 3D feeling of theviewer will be degraded, on the one hand, and the three-dimensionaldisplaying capabilities of the applied 3D multiview display will not beexhausted, on the other hand.

It is therefore an object of the present invention to provide a methodfor generating multiple synthesized images for a multiviewautostereoscopic display, wherein the input stereoscopic images areprocessed so that the depth effect of each image view displayed on themultiview stereoscopic display reach its maximum level with respect tothe capabilities of the particular display.

SUMMARY OF THE INVENTION

These and other objects are achieved by providing a method forgenerating multiple synthesized images for a multiview autostereoscopicdisplay device, the method comprising the steps of:

a) providing a stereoscopic input image composed of a left input imageand a right input image,

b) determining respective pixel pairs of the left and right inputimages,

c) generating disparity information from the respective pixel pairs ofthe left and right input images in the form of a first disparity map,

d) scaling each disparity value of said first disparity map so that therange of the scaled disparity values reaches at least one of theboundaries of a preset maximum disparity range associated with saiddisplay device, thereby generating a second disparity map, and

e) based on said second disparity map, generating a preset number ofsynthesized images for simultaneously displaying on said multiviewautostereoscopic display device, wherein each pair of adjacentsynthesized images presents a synthesized stereoscopic image.

The above objects are further achieved by providing an apparatus forgenerating multiple synthesized images for a multiview autostereoscopicdisplay device, the apparatus comprising

-   -   a means for receiving a stereoscopic input image composed of a        left input image and a right input image,    -   a means for determining respective pixel pairs of the left and        right input images,    -   a means for generating disparity information from said        respective pixel pairs of the left and right input images in the        form of a first disparity map,    -   a means for scaling each disparity value of said first disparity        map so that the range of the scaled disparity values reaches at        least one of the boundaries of a preset maximum disparity range        associated with said display device, thereby generating a second        disparity map, and    -   a means for generating, based on said second disparity map, a        preset number of synthesized images for simultaneously        displaying on said multiview autostereoscopic display device,        wherein each pair of adjacent synthesized images presents a        synthesized stereoscopic image.

BRIEF DESCRIPTION OF DRAWINGS

The present invention will now be described through its preferredembodiments with reference to the accompanying drawings, in which:

FIG. 1 is a flow diagram illustrating the steps of the method accordingto the present invention,

FIG. 2 is a flow diagram illustrating the steps of the adaptiveoptimization performed in the method according to the present invention,

FIG. 3 shows image representations of the intermediate image processingstages in a preferred embodiment of the method according to theinvention

FIG. 4 is simplified schematic view illustrating the overlap areadetection in the method according to the invention,

FIGS. 5.a and 5 b are simplified schematic views illustrating thegeneration of the synthesized image pixel values in a scan line in themethod according to the present invention, and

FIGS. 6.a and 6.b illustrate examples for the relative viewing positionsof the synthesized views with respect to the left and right inputimages.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The basic steps of the method according to the invention are describedwith reference to FIG. 1, which shows a schematic flow diagram of themethod, and FIG. 2, which illustrates the image representations of theintermediate image processing stages based on an exemplary input pair ofstereoscopic images.

The input stereoscopic image of the method is formed by an associatedpair of input images, referred to as left and right input images thatmay be either still images or video frames. In case of video frames, thesubsequent input frame pairs of the video stream are processed anddisplayed in a video stream. The input images may be obtained by astereoscopic photo camera, a stereoscopic video recorder or they may becreated even by an appropriate computer program that generates renderedstereoscopic images with a format suitable for further processing by themethod according to the invention.

As shown in FIG. 1, in the first step S100, the input images 31 a and 31b are subject to a pre-processing, wherein the left and right inputimages 31 a, 31 b are filtered by a noise filtering algorithm to obtainimage intensity information from the input images 31 a and 31 b. Itshould be noted that from the point of view of the method according tothe invention, the image intensity information has higher significancethan the colour intensity information, this latter being more importantin the prior art image processing methods.

In the next step S110, the filtered images 32 a, 32 b are subject torectification to produce rectified images 32 a and 32 b illustrated inFIG. 3. Rectification is used to produce an image pair, wherein thereexist pixel displacements between the left and right images only inhorizontal direction, i.e. the left and right images present a relativedisplacement only horizontally. Rectification of the input image pair isnecessary to allow for the following disparity map generation algorithmto seek only for unidirectional (i.e. horizontal) pixel displacementswith respect to the left and right images. This simplification thereforereduces the required computational power of the rectification process toa great extent, thereby also facilitating the real time imageprocessing, for which there exists a high demand, in particular in 3Dvideo imaging.

Although the rectification and a subsequent search for pixeldisplacements in a single direction might be substituted by abidirectional search for the pixel displacements in the input imagepair, the computational costs are lower in the former case and thereforethat solution is preferred in the method according to the invention.

Rectification is carried out by a specific algorithm that searches forcorresponding pixels in the left and right images. As a result, therectification process provides, for each pixel of the left image, theposition of the corresponding pixel in the right image, wherever it isavailable. Thereby a two-dimensional vector field is generated for theleft image to define the corresponding pixels in the right image bydirected vectors. The right input image is then distorted in Step S120so that only horizontal displacements will be shown in the right inputimage with respect to the left input image. The one-dimensional vectorfield, or displacement field, thus obtained is the so called “disparitymap” in which defines for each pixel of the base image (i.e. left inputimage) a unidirectional (e.g. horizontal) displacement giving theposition of the corresponding pixel in the other (i.e. the right) inputimage. If the displacement has a negative value, the pixel is moved tothe left relatively to the base image, whereas a positive value for thedisplacement means a relative movement to the right. If the displacementis zero, the particular pixel remained in the same position within theinput image pair. In the special case when the value of the displacementis “void”, the particular pixel of the base image does not exist on theother input image. This latter case occurs when a portion of the baseimage of the input image pair becomes covered in the other image ofinput image pair.

A gray-scaled graphical representation of the disparity map for theinput images 31 a and 31 b are illustrated in FIG. 3 as image 33.

We note that for the sake of simplicity, always the left input imagewill be regarded as the base image for the disparity map throughout thepresent description, it is obvious for a skilled person that thedisparity map may equally be generated with the right input image beingthe base image.

Generation of the disparity map in Step 120 may be carried out, forexample, by a windows-based search, wherein for each pixel of the inputbase image (i.e. the left image in this example), a window of N by Mpixels is formed and the same (or minimally different) window is soughtalong the line of the particular pixel (referred to as “scan line”below) within a distance of ±D. Due to the rectification performedpreviously, there is only one-directional, namely horizontal,displacements between the corresponding pixels of the two images of theinput image pair. Hence, a one-directional search is enough to becarried out for the generation of the disparity map. A more detailedintroduction to the above mentioned window-based disparity mapgeneration procedure can be found in the U.S. Pat. No. 6,314,211, thecontent of which is incorporated here by reference and described in thisspecification only with respect to the particular application thereof inthe preferred embodiments of the present invention.

To cover the inherent noises, compression losses and other errors, thevalue of N and M are to be set to high enough, although some care shouldbe taken when adjusting the values for N and M because in case N and Mhad too high values, the search method could not perhaps findcoincidences at the depth boundaries. For the same reason, the value ofD should also be set at least to a value equal to the greatest pixeldisplacement between the two input images.

Concerning the particular values of N, M and D, it is recommended that Nand M be 0.5-1% and D be 5-10% of the width of the image (or the heightof the image in case of a standing image). It is obvious for a skilledperson that both ranges in the foregoing depend strongly on theparticular content of the input image pair. If, for example, the inputimage is blurred, the cited range for N and M might be insufficient. Onthe other hand, if an object shown in the image presents a too intensivedepth effect (i.e. it appears to pop-out from the image plane to anexcessive extent), causing great displacement values for the pixels ofthe object, the above range for the value of D might be insufficient aswell. In order to eliminate these issues, an adaptive adjustment for thevalues of N, M and D may be used, wherein the search is started byapplying the above recommended values and if no identical orsufficiently similar areas can be found with using these values, saidvalues will be increased until the degree of similarity will beacceptable or the maximum values of N, M and D are achieved.

As the disparity map thus obtained probably contains noises or othererrors, these noises and errors should be reduced or even removed fromthe disparity map by appropriate noise filtering and/or error reductionalgorithms carried out in Step 130 shown in FIG. 1, the result of whichcan be seen in FIG. 3 as image 34 in the present example.

Noise filtering may be carried out by using any conventional filteringalgorithm, such as the well-known median filtering. In this context, anarea with highly deviating disparity values relative to its environmentwhile having a negligible surface (i.e. an area having so small surfaceas it cannot be regarded as an individual object at all), is alsoconsidered as noise and thus matched to its environment, for example, byreplacing its disparity values with the average of the its surroundingdisparity values.

The above improved disparity map may further contain overlappedportions, presented by white areas in the image 35 of FIG. 3. Theoverlap areas are defined as areas which can be seen in either of theinput image but cannot be found in the other image of the input imagepair because of being in an overlap position in the other image seenfrom the corresponding point of view. For such overlap portions of thebase image, the search method for generating the disparity map will notobviously be able to find the corresponding image pixels on the otherimage of the input image pair. Consequently, such overlap areas shouldbe filtered out to reduce the error of the disparity map. The result ofthis overlap area detection can be seen in FIG. 3 as image 36.

There are two types of overlap area, one of which is referred to as aleft side overlap area, whereas the other one is referred to as a rightside overlap area. The phenomenon of the overlap areas will now beexplained with reference to FIG. 4.

The overlap area is of the left side type when it is present on the leftinput image, but not present in the right input image as two separateareas with different depths in the left image get closer to each otheron the right image. Such a left side overlap area disappears from theright image. Concerning the disparity values, for which a specificexample is illustrated in FIG. 4, the left side overlap may berecognised in the following way. Assuming that in the positions I and Jalong a scan line of the disparity map, the disparity values are V andW, respectively, where W<V, the difference J−I should be equal to orgreater than the value of the expression W−V+1. Areas that violate thiscondition appear to move toward each other and therefore are regarded asleft side overlap areas. These areas may be treated as noise.

Analogously, the right side overlap area is defined as an area thatcannot be seen in the left and input image, but it becomes visible inthe right input image because two adjacent objects in the left image getfurther from each other on the right image, i.e. a portion of an objectbeing in a covered position becomes uncovered in the right image.Concerning the disparity values, the right side overlap may berecognised in the following way. Assuming that in the position I and I+1along a scan line of the disparity map, the disparity values are V andW, respectively, if the condition V≠W is satisfied, right side overlapoccurs. In this case, the content of the overlap areas in thesynthesized images will be obtained from the right input image. Anexample for the right side overlap is illustrated also in FIG. 4.

In order to produce the highest possible three-dimensional effect in thedisplayed stereoscopic images, the disparity map processed by the abovemethods, resulted in the representation as shown by image 36 of FIG. 3,for example, is further processed by an optimizing procedure in apreferred embodiment of the method according to the present invention.The step of this optimization is illustrated as step S140 in FIG. 1.

The objects of the optimizing procedure include:

-   -   adaptive adjustment of the focal points of the virtual cameras        associated with the synthesized views;    -   according to the adjusted focal points of said virtual cameras,        extension of the displacement values of the disparity map to        cover the widest possible range available for the particular        display in use with respect to the capabilities of said display;    -   modification of the displacement values of the disparity map so        that the distribution of the displacement values approach a        uniform distribution to the highest possible extent without        visible distortion of the image contents with respect to the        original images of the input image pair.

The adjustment of the focal points of the virtual cameras associatedwith the synthesized view is necessary in order to compensate possiblefocusing errors of a real stereo camera, such as a focal point at theinfinity or parallel viewing directions for the left and right cameras.If an image were captured, for example, by a stereo camera with parallelviewing directions for the left and right cameras, the focal point ofthe recorded images would be located at the infinity when a focal pointadjustment of the synthesized views would not be done. In lack of suchan adjustment, the synthesized views would not be imperfect and wouldnot provide a sufficient spatial effect. Adjustment of the focal pointsof the virtual cameras will actually be performed by the following twoissues.

Concerning the second issue above, it is well-known that in all displaydevices there can be defined a specific maximum displacement value,above which the image becomes blurred or spurious to the extent thatcannot be tolerated by the viewer any more. This critical displacementis about 1-2% of the maximum resolution of the display device. As themaximum disparity between two adjacent views cannot exceed this criticalvalue, a maximum displacement value MAXD is defined for the particularmultiview display where MAXD is equal to a pre-set critical displacementvalue multiplied by the number of views presented by the display device.For example, in case a critical displacement of 1% is pre-set in themethod and the display is capable of presenting seven views, MAXD willhave a value corresponding to the 7% of the maximum resolution of thedisplay device. The value of MAXD may be exhausted in both positive andnegative directions since an object accommodating in front of the focalpoint in the depth direction moves to the left in the right input imagewith respect to the left input image (or similarly, to the right in theleft input image with respect to the right input image), whereas anobject behind the focal point moves to the opposite direction. Theposition in the input images where no displacement takes place betweenthe two input images, will be deemed the focal point of the (real orvirtual) stereo camera. All contents behind this focal point arereferred to as “inscreen” contents, whereas all contents in front ofsaid focal point are referred to as “outscreen” contents. In otherwords, inscreen objects appear behind the plane of the display in thedepth direction, and outscreen objects appear in front of the plane ofthe display, i.e. these objects “pop-out” from the display toward theviewer.

The foregoing object is achieved by the scaling of the disparity mapcarried out in Step S141 shown in FIG. 2. In the scaling step, thedisparity values of the disparity map are scaled up to cover the entirerange between −MAXD and +MAXD. The scaled disparity map thus obtained isrepresented by image 37 in FIG. 3.

The third object of the optimization, i.e. providing an approximatelyuniform distribution for the displacement values of the disparity map,addresses to modify the disparity map so that no object appears in thesynthesized views excessively far from or excessively close to theviewer in the depth direction. In case an object were located too closeto the camera, its associated disparity values would not change muchupon the scaling of the disparity map, however, the disparity values ofother objects accommodating farther from this object in the depthdirection might be scaled up to a relatively greater extent, which wouldresult in a relative reduction between the disparity of the particularobject and said other farther objects in the synthesized view, thuscausing a substantial degradation of the spatial or depth effect in thesynthesized views.

One possible approach to achieve the above mentioned approximatelyuniform distribution for the disparity values of the disparity map is toprovide a substantially linear distribution of the depth valuesassociated with the various objects in the image. Obviously, such atransformation of the disparity map results in a distortion of thespatial ratios between the objects visible in the input image, whichmight disturb the viewer. Such a distortion may include the stretch orthe compression of an object with respect to the original size thereofin the input image. To avoid or at least to minimize the visibledisturbing effects of such a distortion, it is preferred that theunimportant intermediary areas between the objects are transformedinstead of the objects themselves. To this end, in step 142, aclassification (or clustering) of the objects in the input image iscarried out before the above mentioned step of linearization. The aim ofthis classification step is to separate the objects in the input imageinto multiple groups depending on the intensity and the depth of theobjects. The result of such a classification step in the example of FIG.3 is represented by images 38.

After completing this classification step, disparity values of therelated objects are scaled up to a relatively lower extent, whereasdisparity values of the intermediary areas between the clustered objectsare scaled up more intensively in step S143 shown in FIG. 2. Anexemplary representation of the disparity map resulted from theforegoing step of linearization can be seen in FIG. 3 as image 38.

By taking the latter two steps, i.e. the classification and thelinearization of the disparity map, the first condition above will bealso satisfied for the following reason. Since the disparity maps arescaled up to extend over the entire range from −MAXD to +MAXD, theremust be disparity values with zero or approximately zero values whichdefine a base plane, wherein objects presented in this base plane appearat a half depth of the entire depth range sensed by the viewer on thedisplay. This base plane is regarded as the plane to which focusing ofthe synthesized views actually takes place. This base plane is commonfor the left and right images of the synthesized image pairs.Consequently, negative disparity values define objects appearing behindsaid base plane embedding the focal point of the virtual cameras,whereas positive disparity values define objects appearing in front ofsaid base plane, i.e. the outscreen content.

Having optimized the disparity map by completing the above steps ofscaling, classification and linearization, a calibration of thesynthesized views are carried out in step S150 followed by generatingthe particular synthesized views for simultaneously displaying on themultiview stereoscopic display device in step S160, as shown in FIG. 1.

In the step of calibration, the number of synthesized views (N) and thevalue of the interpolation displacement (H) may be adjusted by takingthe available features of the display device into view. By varying thevalue of H, the extent of displacement of the synthesized views towardsthe left input image or the right input image can be controlled. Forexample, if H is set to a relatively high value, the synthesized viewsare moved toward the right input image, causing the user to observe theobject of the input image from rather left side.

Upon the calibration of the synthesized views, the particularsynthesized views can be generated. Some exemplary synthesized imageviews are illustrated in FIG. 3 as images 40 a to 40 e. In thefollowing, the step of synthesized image generation will be described indetail with reference to a particular example illustrated in FIGS. 5 aand 5 b.

Assuming that N synthesized views are to be generated, where N≧2, andthe Kth view is the next view to be generated, where 0≦K≦N−1, the pixelvalues of this image view are calculated by the following algorithm.

-   -   1. Each pixel P(x, y) of the left input image is presented at        the pixel position P′(x+D,y) of the Kth view, where        D=(K/(N−1)−H)*DM[x, y] and DM[x,y] is the disparity value at        position (x, y) of the optimized disparity map. This algorithm        determines a linear interpolation with a displacement of H        within a scaled interval of the disparity values ranging between        0 and 1. Due to this transformation the original left input        image is distorted so that the synthesized views appear between        the left and the right input image, or even beyond them as well,        as a result of the above mentioned scaling of the disparity map.        If H=0 and K=0, then the synthesized view will have the same        viewing angle as that of the left-side real camera because D=0        in this case.

As shown in FIG. 5.a, when two synthesized views are generated accordingto the optimized disparity map (i.e. N=2), for K=0 and H=0, thedisparity value of −4 will be transformed to a scaled displacement D=−2,and for K=1 and H=0, the disparity value of +2 will be transformed to ascaled displacement D=+2. The pixel values of the original left imageare displaced by −4 or +2 in the first and second synthesized views withK=0 and K=1, respectively.

-   -   2. In the areas where right-side overlap has been detected, i.e.        areas which can be seen in the right input image but cannot seen        in the left input image, those areas of the right input image        may be used to be added to the undefined areas of the        synthesized views. These pixels are indicated with “R” in the        exemplary synthesized scan lines in FIG. 5.a. Since no pixel        information from the left input image is available for these        areas of the synthesized views, corresponding pixels of the        right input image are inserted therein.    -   3. Due to the scaling-up of the disparity map in the scaling        step, there might also be areas in a particular synthesized view        which cannot be seen in either of the input images because of        the so called overdistortion, i.e. when the particular        synthesized view is generated with a virtual point of view from        which certain portions of the synthesized view could not be seen        from the real viewing point used for the input images. To        provide pixel information for such areas, pixels of the        synthesized view may be interpolated from known adjacent        portions of the input images along the scan line. A particular        example for such a situation is illustrated in FIG. 5.b.

In FIG. 5.b, a portion of a line of the optimized disparity map and thecorresponding pixel intensity values are presented. As can be seen inthe figure, there are three pixels that do not appear in none of theleft and right input images because of the overdistortion mentioned inthe foregoing. Taking the adjacent pixel intensity values V and W, i.e.32 and 54, respectively, in this example, for the undefined range of Npixels in the scan line of the synthesized view, intensity P of the Kthpixel may be defined by the expression P=V+(W−V)*((K+1)/(N+1)), yieldingpixel intensity values 37, 43 and 48 for the three undefined pixels ofthe particular scan line.

Finally, the synthesized views are presented on a multiview stereoscopicdisplay in a conventional manner in Step 170. In FIGS. 6.a and 6.b,examples for the relative viewing positions of the synthesized viewswith respect to the left and right input images are illustrated for amultiview stereoscopic display with five views.

As shown in FIG. 6.a, due to the content (not shown) of the input stereoimage, the five synthesized views 62 a to 62 e are presentedsubstantially within the viewing angle □₁ associated with the left andright input images 61 a and 61 b, resulting in a reduced viewing angle□₁ for the adjacent synthesized views 62 a to 62 e, each adjacent pairsthereof defining a synthesized stereoscopic image. This occurs, forexample, when the stereoscopic input image contains objects that appearto the maximum distance from each other which can be presented on theparticular display. In this case the intermediate synthesized views 62b, 62 c and 62 d substantially appear as interpolated views between theleft and right input images 61 a and 61 b.

FIG. 6.a illustrates a case where the disparity range of the inputstereo image pair is too high for a comfortable presentation to theviewer, and therefore the disparity range of the synthesized image pairsshould be reduced to a lower level that can be displayed to show acomfortable view for the viewer. This reduction of the originaldisparity range may be interpreted as the viewing angle associated withthe synthesized views is smaller than that of the input image pair. Sucha reduction in the disparity range, however, results in a reduced deptheffect as compared to the input stereo image

FIG. 6.b illustrates another example where the synthesized views 64 a to64 e are positioned outside the input views 63 a and 63 b as the viewingangle □₂ associated with two adjacent synthesized views are greater thanthe viewing angle □₂ associated with the left and right images 63 a and63 b of input stereo image. Typically, this situation occurs when theobjects of the input image appear like accommodating within a narrowtransversal space volume in the depth direction, i.e. hardly any spatialeffect can be observed by the viewer in the input stereo image.

The number of synthesized views depends on the type of the display andtherefore this number is an adjustable parameter in the method accordingto the invention. It also preferred that the value of MAXD may be setdepending on the number of views presented by the display device. It isfurther preferred that MAXD is set to a multiple of 1% according to thenumber of views of the multiview display. For example, in case of amultiview display with five views, MAXD is to be set to 5% of themaximum pixel resolution of the display, whereas for a multiview displaywith eleven views, MAXD may be set even to 11% of the display's maximumpixel resolution in order to present three dimensional views with thehighest possible spatial or depth effect that can be reached by theparticular multiview autostereoscopic display.

It should be noted that in the method according to the presentinvention, only synthesized views generated on the basis of the left andright input images are presented in the display and only in specialcases it may occur that any of the synthesized views is identical to anyof the input images. In practice, this is a rather rare situation.Another important feature of the method according to the presentinvention is that the viewing positions of the synthesized view dependon the content of current input image pair as, well as the number of theviews presented by the multiview display, thus a particular synthesizedview may appear between the left and right views of the input image(which are not presented by the display generally) or even outside(beyond) the view of the input image. However, the spatial or deptheffect of the presented synthesized views is at least maintained at thelevel of the input image or may be even amplified to some extentrelatively to that of the input image.

In a second aspect, the present invention also relates to an apparatusfor generating multiple image views for a multiview autostereoscopicdisplay device, wherein said apparatus comprises means for receiving astereoscopic input image composed of a left input image and a rightinput image, means for generating a first disparity map for eachrespective pixel pair of the left and right input images, means forscaling the first disparity map so that the disparity values of thefirst disparity map fit to a preset maximum disparity range determineddepending on the particular display device, thereby generating a seconddisparity map, and means for generating, according to the seconddisparity map, a preset number of synthesized images for simultaneouslydisplaying on a multiview autostereoscopic display device, wherein eachpair of adjacent synthesized images presents a synthesized stereoscopicimage.

Preferred embodiments of the apparatus according to the invention maycomprise further means for carrying out any of the foregoing steps ofthe method according to the present invention. The configuration ofthese embodiments of the apparatus is obvious for those skilled in theart on the basis of the above specification of the method according tothe invention, and therefore those are not detailed herein.

More particularly, the apparatus may further comprise a means forfiltering noise in the left and right input images as mentioned above inrelation to the pre-processing step S100 of the method according to theinvention. The apparatus may also comprise a means for rectifying theinput images to obtain only one-directional displacements in either ofthe left and right input image with respect to the other one. Such ameans may be used in step S110 of the method according to the inventionand may also be adapted to perform the rectification in the stereoscopicinput image in the horizontal direction.

The apparatus may further comprise a means for filtering noise andreducing errors in the first disparity map in accordance with step S130of the method according to the invention. Preferably, the apparatus alsocomprises a means for the classification of the objects in thestereoscopic input image and a means for the linearization of thedisparity map to produce said scaled second disparity map to performsteps S142 and S143 of the method according to the invention,respectively.

In a particularly preferred embodiment, the apparatus may furthercomprise a means for calibrating the synthesized views for thegeneration thereof, said calibration means including a means foradjusting the number of the synthesized views to be generated, as wellas the value of the interpolation displacement. Such a calibration meansmay be used in step S150 of the method according to the presentinvention.

The apparatus according to the invention is preferably adapted toreceive a stereoscopic still image, a stereoscopic video frame or astereoscopic rendered computer image as a stereoscopic input image.

Although in the foregoing, several preferred embodiments of the methodand the apparatus according to the invention have been illustrated, thepresent invention is not in any way limited to the exemplary embodimentsshown in the description and the drawings and many variations thereofare possible within the scope of the invention defined by the appendedclaims.

1. A method for generating multiple synthesized images for a multiviewautostereoscopic display device, the method comprising the steps of: a)providing a stereoscopic input image composed of a left input image anda right input image, b) determining respective pixel pairs of the leftand right input images, c) generating disparity information from therespective pixel pairs of the left and right input images in the form ofa first disparity map, d) scaling each disparity value of said firstdisparity map so that the range of the scaled disparity values reachesat least one of the boundaries of a preset maximum disparity rangeassociated with said display device, thereby generating a seconddisparity map, and e) based on said second disparity map, generating apreset number of synthesized images for simultaneously displaying onsaid multiview autostereoscopic display device, wherein each pair ofadjacent synthesized images presents a synthesized stereoscopic image.2. The method according to claim 1, wherein the method further comprisesa step of filtering noise in the left and right input images.
 3. Themethod according to claim 1, wherein the method further comprises a stepof rectifying the input image to obtain only one-directionaldisplacements in either the left input image or the right input imagewith respect to the other one.
 4. The method according to claim 3,wherein rectification is performed in the input images in the horizontaldirection.
 5. The method according to claim 1, wherein the methodfurther comprises a step of noise filtering and error reduction in thefirst disparity map.
 6. The method according to claim 1, wherein themethod further comprises a step of classification of the objects in thestereoscopic input image and a step of linearization of the disparitymap to produce said scaled second disparity map.
 7. The method accordingto claim 1, wherein the method further comprises a step of calibrationof the synthesized images for the generation thereof, said step ofcalibration including the adjustment of the number of the synthesizedimages to be generated and the value of the interpolation displacement.8. The method according to claim 1, wherein the stereoscopic input imageis a stereoscopic still image, a stereoscopic video frame or astereoscopic rendered computer image.
 9. An apparatus for generatingmultiple synthesized images for a multiview autostereoscopic displaydevice, the apparatus comprising a means for receiving a stereoscopicinput image composed of a left input image and a right input image, ameans for determining respective pixel pairs of the left and right inputimages, a means for generating disparity information from saidrespective pixel pairs of the left and right input images in the form ofa first disparity map, a means for scaling each disparity value of saidfirst disparity map so that the range of the scaled disparity valuesreaches at least one of the boundaries of a preset maximum disparityrange associated with said display device, thereby generating a seconddisparity map, and a means for generating, based on said seconddisparity map, a preset number of synthesized images for simultaneouslydisplaying on said multiview autostereoscopic display device, whereineach pair of adjacent synthesized images presents a synthesizedstereoscopic image.
 10. The apparatus according to claim 9, wherein theapparatus further comprises a means for filtering noise in the left andright input images.
 11. The apparatus according to claim 9, wherein theapparatus further comprises a means for rectifying the stereoscopicinput image to obtain only one-directional displacements in either theleft input image or the right input image with respect to the other one.12. The apparatus according to claim 11, wherein said means forrectifying the stereoscopic input image is adapted to perform therectification in the stereoscopic input image in the horizontaldirection.
 13. The apparatus according to claim 9, wherein the apparatusfurther comprises a means for filtering noise and reducing errors in thefirst disparity map.
 14. The apparatus according to claim 9, wherein theapparatus further comprises a means for the classification of theobjects in the stereoscopic input image and a means for thelinearization of the disparity map to produce said scaled seconddisparity map.
 15. The method according to claim 9, wherein theapparatus further comprises a means for the calibration of thesynthesized images for the generation thereof, said calibration meanscomprising a means for adjusting the number of the synthesized images tobe generated and the value of the interpolation displacement.